Strong Truthfulness in Multi-Task Peer Prediction

The problem of peer prediction is to elicit information from agents in settings without any objective ground truth against which to score reports. Peer prediction mechanisms seek to exploit correlations between signals to align incentives with truthful reports. A long-standing concern has been the possibility of uninformative equilibria. For binary signals, the Dasgupta-Ghosh output agreement (OA) mechanism [2] leverages reports across multiple tasks to achieve strong truthfulness, so that the truthful equilibrium maximizes payoff. In this paper, we first characterize conditions on the signal distribution for which the OA mechanism remains strongly-truthful with non-binary signals. Our analysis also yields a greatly simplified proof of their binary-signal, strong truthfulness result. We then introduce the 01 mechanism, which extends the OA mechanism to multiple signals, with a slightly weaker incentive property: no strategy provides more payoff in equilibrium than truthful reporting, and truthful reporting is strictly better than any uninformed strategy (where an agent avoids the effort of even obtaining a signal). In an analysis of peer-grading data from a large MOOC platform, we investigate how well student reports fit our model, and conclude that the 01 mechanism would be appropriate for use in this domain.